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PREFACE 


This paper was originally intended to be a simple guide to the preparation of a 
programme of training for Census Enumerators and was prompted by experience as 
training advisor to the Central Statistics Office in Kabul for the first full 
Population Census of Afghanistan which was taken in June/July 1979. Further 
experience and discussions with census takers from other countries - notably Thailand 
- led me to conclude that a broader approach to the subject should be made 


particularly because of the wide interest being shown in my proposal. 


The result is a paper which, to those well versed in modern training techniques is at 
times a statement of the obvious whilst to those responsible for the management of a 
census it may possibly appear impractical.e To the former I can only beg indulgence 
for what must, in this paper, be both an over simplification and a somewhat sketchy 
outline of the development of a training programme and all its stages. To the latter 
I will refer to the enormous cost and effort needed to organise and carry out a 
successful census - generally acknowledged to be the largest peacetime operation 
which a nation undertakes - and to underline the importance of an effective training 


programme. 


As will be seen later a training programme for census field staff is unique and I 
have been fortunate to have had experience of both training and the management of 
censuses in several counties sufficient to appreciate the huge gap which often exists 
between the two and, further, to recognise the difficulties in trying to reconcile 


them. 


As a permanent employee of the Office of Population Censuses and Surveys (for England 
and Wales) I am obliged to point out that the views expressed in this paper are 


entirely personal and not necessarily shared by that Office. 


I would like to record my thanks to the Office - which, incidentally I firmly believe 
is firmly established in the forefront of census methodology - for allowing me to 


publish this paper. 


Tony Rampton 
November 1979 
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Part | - Introduction 


1.1 


1.3 


A Population Census is a fact finding operation with statistics as its end 
product. As these statistics play an important part in the planning of a 
nation's future it is essential that they are complete and accurate and, that 
they are standardised as far as possible throughout the country so that direct 
comparisons can be made between, say, one area and another or between particular 
groups of people. To achieye these aims it is a primary requirement that all 
field staff - Enumerators especially - follow the same laid down procedures but 
to accomplish this is no easy matter. It must be remembered that Enumerators 
comprise an exceptionally large group of employees over whom, during the 
enumeration itself, little or no control can be exercised. They are also 
employed for the specific task of enumerating the population and are drawn from 
the public at large, usually government employees or teachers, and having no 
real knowledge of the census and its requirements. Wrong actions at the data 
collection stage may lead to poor quality or incomplete information, perhaps 
even to undercounting of the population the result being inadequate or 
misleading statistics leading to government policies based on false premises. 
To lessen this risk it is clear that Enumerators must be thoroughly conversant 
with all their duties, with definitions, basic and special enumeration 
procedures and with any special requirements of the census planners. It follows 


that training of the highest quality is of the utmost importance. 


The life-span of Census Enumerators is brief, however, and it is no easy task 
to equip them with the necessary knowledge, to develop their skills and to 
motivate them for such a short, one-time operation as the census enumeration. 
Nor is it easy to ensure they are all trained to the same, uniform level 
throughout the country. Yet this is precisely what must be attempted. There is 
no second chance for Enumerators; once trained they must carry out almost all of 
their duties unsupervised. There is seldom the opportunity to arrange remedial 
tuition once the field operation has started and even less chance of putting 


things right even supposing that deficiencies can be detected. 


An added complication is that field trainers are usually remote from the chief 


planners at Census Headquarters and must be trained at second or even third 


hand. However skilled they may be in the techniques of training they will 


almost certainly lack the detailed knowledge of the census and its requirements 


ith 
and in deciding what needs to be taught. Indeed, the very nature of census wit 


its need for uniformity decrees that field trainers must not be permitted to 


decide for themselves. Training content and, preferably, methods must be 


to 
decided at the highest level of census management and steps must be taken sai 


ini chaine 
ensure proper implementation at each successive level down the training 


{ 


1.4 Census field staff form a pyramid with Headquarters staff - managerial, 
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technical and training experts - at the top, and at least two, often more, 
levels below. In Afghanistan, for example, a Provincial Office was established 
in each of the 28 Provinces with a further 150 District Offices for more local 
controle Field staff comprised about 400 Supervisors, 2000 ‘Controllers and 
10,000 Enumerators of whom about 10 per cent dealt with the enumeration of the 
Nomadic element of the population. The training programme was planned for 4 
levels with Census Office specialists instructing Provincial Office staff 
direct. At the next level down District Office staff and Field Supervisors were 
trained together by Provincial Office trainers and they in turn trained, first, 
Controllers and then Enumerators. At the higher levels instruction was given in 


supervisory and organisational duties as well as for the enumeration itself. 


“sy 


This paper attempts to set down in much simplified form some notes about the 
three main stages of a field training programme - its development, 
implementation and evaluation. It is not intended to be a model to be copied 
exactly; rather it is more of a guide for modification as befits existing 


circumstances, planning requirements, resources and financese 


Part 2 - Development 


2-1 


222 


2-3 


Most training programmes benefit from constant repetition, review and revisi 
on 


so that, in time, a high state of refinement is attained. Census on the other 


hand, provides only one real opportunity and the training must be effective on 
the first and only occasion it is given; there is no second chance. It follows 
that successful training can be achieved only by thorough and detailed planning, 
by conducting small scale trials to detect and remedy inconsistencies and to 


strengthen weaknesses. 


To allow for the proper development of the programme, decisions must be made on 
time and sufficiently early by management and subject matter experts. Usually, 
it is considered sensible to establish a ‘technical committee' composed of 
representatives from appropriate disciplines and including the Training 
Administrator for the projecte Development will almost certainly take months 
rather than weeks particularly if the programme also includes the drafting and 


production of basic instruction manuals as well as training requirements. 


In every census the first main priority must be to establish a main framework. 
Concepts and definitions, topics and the questionnaire design, type of census 
field organisation must all be agreed if only in outline as early as possible to 
enable more detailed planning to be started. An overall planning timetable is 
essential. It is particularly important to settle enumeration procedures and 
responsibilities of field staff early in the planning stage so that work can 
begin on the drafting of instruction manuals and the development of the training 
programme. The several stages in this work are explained briefly below and 
whilst each step appears to follow in sequence it should be noted that this is 
rarely the case in practice. Usually there will be a good deal of overlap with 
each successive stage exposing areas of uncertainty in the preceding stage 
until eventually when the training programme has been tested and finally 


approved all defects should have been remediede 
Describe jobs of field staff 


Detailed lists of tasks must be produced for each grade. An example, based on 


Afghanistan's requirements, explains what is needed: - 


The enumerator's job before enumeration is to - 


e attend training course, complete assessment test and collect supplies 


e travel to area of enumeration 


{ 


e meet “important people” and explain reason for visit 


e check enumeration district boundaries 


e check “summary pre-list" 
e plan "work programme" and complete form for Controller 

e prepare materials for enumeration 

e read through instruction manual again before starting the enumeration 


During the enumeration it is to - % 


e visit every listed address 


e@ look for and visit any unlisted addresses 
@ contact a "responsible person" and explain purpose of visit 
e identify "households" and record details on houselist 


@e obtain census information from informant and complete "household 


questionniare" 
@ complete houselist 
e check for any additional households 
@ re-visit any households not contacted at first attempt 
@ record information about "non=-responding households", etc 
e keep records of progress, expenses, etc (daily) 
hand over completed forms to Controller (as arranged) 


(NB The above list is not necessarily comprehensive nor complete and serves 
only to illustrate. ) 


2-4 Analyse and specify details of tasks 
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Training aims at taking people with a varying range of knowledge, skills and 
attitudes to one specific goal - satisfactory terminal behaviour or job 
performance. Unless this can be measured training can be of only limited 
effectiveness. It follows that standards must be set and tested and to do this 
it is necesary to give precise statements of what the student will be required 
to doe. It may be that certain tasks will be well within the existing ability of 


the student in which case there is no need for training but this can be 


ascertained only by careful analysis. 


Each task as previously described must be specified in objective terms and 
further described and progressively re-described to an appropriate level. This 
level can often be decided by considering the effect of failure, or cost, to be 
overall scheme. If the cost in terms of damage to the whole is negligible then 
likelehood of failure does not matter. Nor does it matter if the "cost" is high 
but likelihood of failure practically non-existent. But high cost combined with 
high risk points to the need for further analysis and sub-division into a series 
of sub operations. In this was, instructions can be concentrated on the 
detailed and difficult components of a task and, as a result, become more 


effective. 


For example, the task of "changing the wheel of a car" to a skilled mechanic 
will be simple but for those without the necessary knowledge or skill it needs 
breaking down into a series of simple steps such as "locate jack and raise car", 
"remove wheel nuts", etc. If these simpler tasks are still considered beyond 


the trainee's ability they must be even further analysed. 


Draft instruction manuals 


Having analysed the tasks - bearing in mind that these are the requirements of 
the census planners - a first draft or written statement of the entire job can 
be produced. In effect this is the basis for the field manual which, at this 
stage need only be a series of directives.e Further explanation, style and 
illustrations are important but can be dealt with latere 


It is now imperative at this stage that the technical content of the written 


; i i etin 
instructions is agreed with the subject matter specialists. A special me g 


; i draft 
of the census technical committee is probably advisable and copies of the dr 


should be made available to members for prior examinations 
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The second draft should be more or less in final format complete with 


illustrations. If the printing schedule permits it should then be set aside 


until after trials of the training programme have been completed. Such trials 


. 


often reveal faults in the instructions. 


Set training objectives 
set tra’ 


Readers well versed in training techniques will be aware of the need for setting 


clear objectives but as this paper is written for those less experienced a brief 


explanation may be appropriate. 


A training objective is a statement in precise and unambiguous terms of what the 
learner will be able to do as a result of {eerie A student's knowledge or 
skill is measurable only in terms of his performance. Training helps to Le 
increase knowledge, to develop skills and to modify attitudes as a result of 
which the learner should be able to comprehend and do things which he could not 


do before. In other words his "terminal behaviour" will be different. 


With this in mind it is clear that a training objective must set out in 
behavioural terms what the learner will be expected to do, the conditions under 
which the task will be performed and the standard of acceptable performance. It 
should also include terms which will allow some measure of assessment to be 
made. Clearly an objective which states simply that "After training the | 
enumerator will know what to do during the enumeration" is too broad and loosely 
worded to be useful. What exactly is he required to do and how can his 
performance be measured? Let us take an example. An enumerator will need some 
knowledge of census, its purpose, importance, organisation and so on. Clearly 
he will not be expected to discuss demographic theory at professional level, 
nor to lecture to other trainees. He must, however, possess sufficient 


knowledge to be able to explain the need for a census to people he encounters 
during the enumeration. 


Perhaps the objective is better stated as - 


At the end of Training the enumerator will be able to perform 


the following tasks to the satisfaction of the trainer: - 


explain in his own words and from memory the need for a periodic 


population census and its importance 


give satisfactory answers to the most commonly asked questions by 


members of the public (as listed in the instructions) and without 


reference to the manual 


Other objectives likely to be included in a training programme for census 
enumerators are: = 


@ decide which of the people living at an address (dwelling, housing unit, etc) 
‘ 


are members of the same "“household(s)" 


e@ ask questions as specified in the instruction manual so as to obtain the 
information about household members and their living accommodation as 


required for the Census 


e@ record answers to the questions on the prescribed form(s) legibly and 


accurately 


Setting training objectives for census field staff is not always. easy but it is 
important that it should be done. Unfortunately, because of presure of time or 
lack of early decision it is not always completed and the task of the training 
administrator becomes extremely difficult and the training programme suffers. 
Many thousands of enumerators are employed in a census. Each is required to 
carry out a series of tasks in the enumeration with a minimum of supervision and 
the scope for errors is enormous. Performance can be improved mainly by 
effective training and this can be achieved only if it is done properly and this 


is possible only if the student's precise terminal behaviour can be specified. 


As Robert F Mager says " eeseeeee if you're not sure where you're going, you're 
liable to end up some place else - and not even know it." ("Preparing 


Instructional Objectives" ) 
A simple summary for the preparation of objectives is:- 


1¢ Identify the terminal behaviour and specify the type of performance which 


will be accepted as evidence that the student has achieved it. 


2. Further define the terminal behaviour by describing the important conditions 


under which the student will be expected to perform. 


3. Specify the criteria of acceptable performance to show how well the student 


will be expected to perform. 


pEem-NO0 
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2.7 Devise tests 
SS . 


The construction of a test is often a help in setting objectives. Indeed it can 


sometimes expose weaknesses in them. The most commonly used test is the written 
questionnaire and here again it is useful to agree its content with the 
technical experts. Great care must be exercised in drafting the questions to 


avoid ambiguity and to lessen the risk of incorrect assessment or "marker bias". 


Evaluation of training, as will be emphasised later in Part 4 of this paper, is 
a highly specialised subject best left to experts and as such will not be 
discussed here save to point out the dangers of having the question paper 
drafted by somebody not familiar with the subject. 

A particularly useful purpose served by the test, and one which regrettably is 
often overlooked, is its advance application to a small sample of the target 


Py. 
population. A high success rate at this stage may indicate that certain aspects 


of the training is not necessary. 
2.8 Decide training needs 


Deciding what needs to be taught is perhaps the most diffciult stage in the 
development of a Census training programme. Re-stating the problem —- that of 
training a huge target population with varying levels of knowledge, ability and 
motivation to carry out what is, in practice, a series of relatively simple 
actions but which combine to present an apparently complex operation - it can be 
seen that the temptation is to compress everything into the training course. 

The difficulty is in deciding what can be ignored and what is essential. Which 
of all the definitions, procedures, etc would be most damaging to the census 


results if large number of enumerators “get them wrong"? 


The subject matter expert will say that everything is important and certainly, 
with the low level of knowledge of the target population which must be assumed, 
it is hard to disagree with that view. But the time allowed for training is 
often too short and decisions have to be made. Yet again the training staff 
would be advised to work closely with the technical experts to decide what 
knowledge and skills are needed by an-enumerator to collect data which is 
complete and accurate. Setting these requirements against those that the 
trainees are likely to possess already should throw into relief their training © 


needs. A useful Starting point is to decide not what the student needs to know 


but what he does not know, what is not understood or what is wrong with his 


present performance. 


ye 


2.9 Determine teaching strategies 


Training methods are many and varied. Demonstrations, role play, practical 
’ 


self instruction and others are all bona 
fide techniques and in the normal course of events the training manager will 


work, tutorials, discussion groups, 


select a strategy appropriate to the particular training need. Regrettably the 


nature of the census operation somewhat precludes certain methods. For example, 
discussion among trainees can help modify attitudes but without skilled 
leadership there is a danger that the group may alter course and reach 
conclusions contrary to Pequired objectives. Perhaps it is for this reason that 
census training too often in the past has consisted of a series of lectures 
amounting to little more than a reading of extracts from field instruction 


manuals. This sort of "training" can never be really effective. 


Selection of an appropriate strategy usually depends on the training needs and 
the target population. Clearly both training objectives and methodology will 
not be the same for a group of managers as for operatives. Whereas, the former 
will need to acquire knowledge and comprehension of a wide range of topics 
together with skills necessary for the organisation and supervision of staff the 


latter may need to develop reflex skills for manual tasks. 


A training programme for census field staff must also take account of the needs 
of the operation and the ability of students but the special logistical problem 
of census and the enormous number and wide range of trainees must also be given 
full consideration during its planning stage. It is necesary too to consider 

- the role of the enumerator who during the enumeration will be unsupervised and 
who, at times must be a communicator, cartographer, data collector and more. In 
general, strategies must be selected with the broad aims of giving instruction 
in the basics of the enumeration and in motivating enumerators for the task 
ahead. Every opportunity should be taken to increase their knowledge and the 
more palatable this becomes the greater the chance of success. Classroom 
training is an integral part of a training programme; it is by no means the 
whole. The education of field staff begins with their first contact with census 
perhaps with an explanatory leaflet which can be handed to them when first 
contacted as prospective employees. At the very least, enumerators must be 
issued with their manual of instructions in time for private study before group 
training and, when handed over, the opportunity should be taken, if at all 


possible to give verbal guidance however brief it may be. If the manual can si 


supplemented with some sort of self instructional, or programmed, text so much 


the better. Trainees can then study the requirements of the job at their own 


speed and with no pressure on them. The classroom training session may then 


: ity to 
become more an extension of the self education stage providing an opportunity 


more importantly, to provide far _ 


eded- 


. resolve queries from what has gone before and, 


more practical training thus developing the skills which are ne 


2.10 Develop training materials 
e Deve.) 


The classroom training of enumerators must be delegated to field trainers 
usually numbering hundreds sometimes thousands and it is a virtual certainty 


that among them will be some who will be weak links in the training chaine As 


previously stated, left alone to train enumerators as they think fit these 
trainers could provide training varying widely in content and quality throughout 
the country. The problem is best solved by the use of a standardised "training 
package" which if properly constructed, can provide the next best thing to 
direct training by staff at census headquarters. The package must be designed 
by training experts well acquainted with the practice of giving instruction and 
in consultation with subject matter specialists. Through its use planners will, 
in effect, control not only what is taught but how it is taught. In this paper 
it is not appropriate to describe the detailed development of a training package 
but it will probably follow the accepted method starting with teaching points 


and going through to devising visual and other training aids. 


Its scope and content will depend on the amount of time which can be devoted to 


training in the field but will usually comprise: - 


e the trainer's guide, either a set of comprehensive instructional notes with 
guidelines or a fully scripted (verbatum) manual containing “stage 


directions" for the instructor 


e@ visual aids, usually provided conveniently as a set of "flip charts" 


@® practical exercises 
e@ handouts or additional training material for use by trainees 


A comprehensive package will cover the duties of all field staff and not just 
those of enumerators. This is best achieved by producing a basic package 
designed for enumerators but which can also be presented to the higher levels of 
field staff. Managerial ana Supervisory work can be covered by a series of 
Supplements to the main package and designed more as guidelines for instructors 
and with less emphasis on training technique. Contents of the basic package 


will vary with the needs of the census but may be modelled along the following 
lines: - 


Part 1 Course opening 
pathin SA Od dL 


(Welcome to trainees, review of course content, domestic 


arrangements -modified to suit level of trainee) 
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Part 2 About the census 


(Need for enumeration, requirements and results, field oganisation 
‘ 


communications, etc) | 


Part 3 Basic information 


(Pre-listing stage, maps and other materials, basic concepts, 


definitions, etc) 


7 


Part 4 Enumerator's job 


(preparatory work, main stage requirements, additional tasks) 
Part 4 will probably need to be divided into a number of separate sessions 
dealing successively with common and less common situations and including 
practical demonstrations and student participation. 


Remainder of the package may comprise;=- 


Supplement A - Controller's duties 


Supervisor's duties 


District office activities 


Provincial office activities 


Field liaison and administration 


Using the package and training technique 


aq a a o @G- & 


Assessment test 


Simulate, validate and revise training package 


Whilst the final testing is best carried out in conjunction with the pilot 
census it is beneficial to organise one or more small ad hoc trials beforehand. 
It is essential that subject matter specialists - demographers, cartographers, 
communicators, etc - participate fully in these tests to verify that their basic 
requirements are met and that preparation is complete. Informal “dry runs" may 
also expose faults, omissions and weaknesses in both the training package and, 
possibly, in the written instructions. Naturally, these can be rectified before 


bulk printing and production begins. 


Preliminary trials are best conducted by the training staff with other 


specialists and, possibly, census administrators acting as guinea pigs but in 


the final test, trainers and trainees should, if possible, be selected randomly 


from the target population. At this stage the compilers of the package become 


bserverse Tempting though it may be there must be no interference by them 
re) ° 


since any extra training will distort the results of the evaluation. 


Observation itself is a valuable means of assessing the effectiveness of training 
but more objective measurement is essential. A particularly useful way of 
assessing knowledge gain is through the use of the pre/post test in which 
trainees complete a test paper prior to training and again after it. This 

method can be used also to check on the effectiveness of instruction manuals 
assuming they are available for study prior to the training course. For this 


trainees may be divided into 3 groups and assessed as follows:- 


Group 1 Group 2 Group 3 
Pre-test Pre-test ie Read manual 
Read manual Read manual Pre-test 
Post-test Attend training Attend training 
Attend training Post=-test Post=-test 


It will be seen that Group 1 will measure the value of the manual alone; Group 2 
the value of both manual and training combined; Group 3 will help check the 
value of the training and also, if the appropriate questions are included, 


indicate any adverse effect of a "complex" manual on the morale of the trainees. 


The final stage of development will include any revision or strengthening 


prompted by evaluation. 


dy 


Part 3 - Implementation 


3.1 


3-2 


3-3 


All too often the field training stage of Census and other large scale surv 
: ose: : eys 
with their inherent logistical and apparently insuperable difficulties is paid 


scant attention. Great efforts may be made in the development stages to ensure 


an effective training programme but little or no guidance is given to field 
trainers about how to train, accommodation and so on. At the highest level the 


trainin i i = gees ; 
g will invariably he conducted in ideal conditions. Trainers will 


doubtle: i ; 
ss be experts and accommodation above average but when the training "goes 


live" the picture is likely to be very different. Classes will be too large; 


trainers unfamiliar with the subject and materials; accommodation too small and 


often only barely adequate. 


That these problems exist and are not easy to solve is not in dispute but 
planners must be aware of them and make allowances. At the same time there is a 
need for greater discipline and to strive for the best possible training 


Situation. Field trainers must be encouraged to aim for the highest level. 


It is now generally accepted that an experienced trainer can effectively control 
a group of 10 to 15 trainees. With more people the trainer becomes less able to 
monitor each one's progress and deal with any individual's problems. When very 
large numbers are dealt with real "training" is no longer possible and becomes 


little more than a simple "briefing". 


An obvious solution to the problem would be to increase the number of trainers 
and reduce the size of classes but this may not be easy nor even desirable. 
Alternatively, more than one session could be held so that, for example, instead 
of coping with 60 at once, 4 groups of 15 could be trained. The objection here 
is that the training period will be longer but it is worth considering that a 

2 day intensified course for 15 people may be no less effective than an 8 day 


course for 60. On the contrary it may be a great deal more SO- 


There are other possibilites, among them the combined group/syndicate approach 
in which background and general information is given to the entire class and 
detailed instruction is given to small syndicates by the trainer and several 


assistants (crew leaders, assistant census officers, etc). 


Another reason for smaller classes is that suitable training accommodation will 


more likely be available. Large numbers usually mean cramped and uncomfortable 


conditions hardly conducive to the learning process. Field trainers should be 
given clear guidance about the sort of accommodation to seek and warned of the 


dangers of accepting anything less than a specified standards 


ning which can be given to census 


d be seized. It isa 


Because of the limited amount of trai 
enumerators every possible means of educating them shoul 


lost opportunity, for example, if enumerators are given their instruction 


manuals at the training course and not beforehand. If they can be issued in 


advance - assuming they are studied - trainees will all start their training at 


a common level and many unnecessary questions and time wasting will be avoided. 


There is, perhaps, much scope for the self training of field staff during the 


pre-training period but local conditions will naturally dictate the extent to 


which it can be introduced. 


The timing of training is vital in the census operation. Ideally each level of 


field staff should move directly from the classroom to their work or with as 


little delay as possible. 


A suggested timetable for the training program is as follows: - 


Census training program 


Level Course content Trainers 
TA Introductory course (for Provincial Office staff) Census training staff 
Background to census (assisted by subject 
Provincial and District Office work matter specialists) 


Field organisation 


Administration 


1B Technical course (for PO staff) (As above) 
Enumeration requirements 
Using the training package 


Training District Office staff, Supervisors and 


Controllers 


2 Technical course (for District Office staff Provincial Office 


and Supervisors) staff 


Enumeration requirements 
Using the training package 
Training Controllers 


District Office and Supervisors duties 


3 Technical course (for Controllers) District Office staff 
Enumeration requirements (assisted by 


Controllers duties Supervisors) 


4 Technical course (for Enumerators) District Office staff 
sta 


Enumeration requirements (assisted by 
Supervisors 


and Controllers) 


3-6 This programme provides for two separate courses for trainees at the highest 
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level since they will usually have administrative responsibilities which may 
overshadow the requirements of the enumeration and the training needs of lower 


grades. 


Each successive level of trainee receives training in the requirements of the 
enumeration with additional instruction in their own work and in the work of any 
subordinates they may be required to supervise. One particularly useful stage 
is the training of Controllers by District Office staff which provides an 
opportunity for the trainers to rehearse the package before giving live 


instruction to Enumerators. 


The field training stage should not be unsupervised. It will, naturally, not be 
possible for every training session to be observed but trainers must be aware 
that a visit by somebody from census headquarters is a possiility.e This is not 
only a means of motivating him to greater efficiency but also helps to boost 
morale. Such visits are at first viewed with some suspicion by field trainers 
but, provided the observer adopts the right attitude, are invariably considered 
after the event to have been a highspot. There is a real need to infuse all 
field staff with enthusiasm and team spirit. Field trainers and observers can 


play an important part in fostering these desirable attitudes. 
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Part 4 - Evaluation 


4.1 


4.2 


Evaluation has been defined as "The assessment of the total value of a training 
system, course or programme in social as well as financial terms". Evaluation 

differs from validation in that it attempts to measure the overall cost benefit 
of the course or programme and not just the achievement of laid down objectives. 
The term is also used in the general judgemental sense of continuous monitoring 


of a programme or of the training function as a whole. 


In the context of Census it may be argued that to assess the value of a field 
training programme is too difficult and, in any case, unnecesary because of the 
one-off nature of the exercise. There is no immediate chance or necessity to 
improve so why bother? It might also be considered that validation will have 
been incorporated in pre census tests and the Pilot Census and that, again, it 
is not necessary. But tests cannot replicate the sheer size of the operation 
and, surely, not to evaluate the programme given the comparatively low cost is a 


missed opportunity to learn lessons for the future. 


As emphasised earlier in this paper a good training programme must have clearly 
defined objectives and, if these are not achieved at the main stage, training 
becomes a pointless and costly activity. One way to confirm whether or not 
objectives have been achieved is by means of a properly constructed programme of 
assessment. From this it should be possible to identify weakneses in the 
programme itself and thus provide for improvement in the future.e But there are 
other advantages in such a programme; the assessment of the performance of 
individuals so that poor quality trainees can be identified and replaced; a 
measure of their collective performance through which not only the programme as 
a whole can be scrutinised but also the effect of a large number of field 
trainers and their influence on trainees and, on the data collected; and, to an 
extent, to motivate trainees. These aspects need further explanation but, 


first, let us consider ways and means of implementing an evaluation programme. 


In normal on-going training programmes involving experienced trainers there is 


much to be gained by subjective evaluation particularly in the short term. 


Observation of trainees, the use of questions, verbal tests and two-way 


communication are all valid methods but, generally, are best left to experts 


with both thorough knowledge of the subject and skill in trainings The census 


programme must rely on field trainers who may have one or other of these 


attributes but rarely both and asessment by more objective methods 1s needed. 


: ; i nd 
One obvious way is to inspect the data collected but this is time consuming and, 


: tiveness oa 
often, unrewarding. It is, in any case, not a direct check on the efec 
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of training because field staff may perform badly despite good instructio 
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4.3 


Probably the most productive and convenient method of assessment is the test 


paper mentioned earlier. The design of such a test cannot be thoroughly dealt 


with in this paper but it must be emphasised that preparation is all important. 


Too many tests have proved useless simply because they have been badly designed. 


Question wording has been imprecise or ambiguous; open questions have caused 
"marker bias" making comparisons between groups meaningless; and SO One The 
point must be made. If an objective test paper is to be used it must be 

designed by an expert and itself be tested to ensure it is fault free as far as 


possible. 


Usually the test paper will be administered to trainees at the end of the course 
or programme. It will probably contain a mixture of different types of question 
- yes/no, multiple choice, true/false - there are other typese To help in the 
marking and to ensure standardisation a model or answer key will have been 
supplied for use by the training staff. Immediately after marking it becomes 
possible to identify the poor performers and to dismiss them if they are 
particularly bad. But there is another action which can be taken which is to 
record the results of all trainees in tabular form and in so doing to confirm 


that the training has been properly done. Consider the following simplified 


table of test results: = 


Trainee Number All 
Question 1 2 3 4 5 trainees 
Number 

1 / X vA / rs 4 

2 / X rd z X 3 

3 / X / / Z 4 

4 X X ».4 X X = 

5 / X / i / 4 
All 
Questions 4 = 4 4 3 15 


Such a table would usually cover many more trainees and questions but this will 
serve to illustrate. Clearly trainee number 2 has failed badly. He has got 
some questions wrong which all other trainees have answered correctly and there 
is no doubt that his performance is not acceptable. three trainees (1, 3, and 
4) scored well and the fifth dia reasonably well. But look now at the other 
axise Question 4 was answered wrongly by all trainees and the chances are high 


that either the training was wrong or the trainer did not teach. 


4.4 


4.5 


So by tabulating results in this way it becomes possible for trainers themselves 


to assess results and to spot faults which could be remedied before it is too late 


The same technique can be extended by aggregating marks to enable direct 
comparisons to be made between groups of trainees. At this level results can be 


used to assess the efficiency of the trainer. In the following table there are 


marked differences in the performances of the 2 groups of trainees. In Group A 


only a third of all trainees had error rates of less than 10 per cent; in Group 


B more than twice as many were as successfull. In Group A over 30 per cent 


Percentage No of trainees Per cent Cumulative per cent 


errors A B A B A B 


Less than 10 8 V7 32.0 68.0 32.0 68.0 

10-19 9 6 36.0 24.0 68.0 92.0 

20-29 4 2 16.0 8.0 84.0 100.0 

30-39 2 - 8.0 - 92.0 - 

40-49 1 - 4.0 - 96.0 - 

50 and over 1 - 4.0 ~ 100.0 - 

ES CE ie Ae er Sree, ee nan eee eee en ES Sed 

Total 25 25 100.0 100.0 - - 


of trainees had error rates of 30 per cent or more compared with only 8 per cent 
of trainees in Group B. It is, of course, not possible to be certain where the 
faults lie - with trainees, the training package or the trainers - but some 


indication may be given in the next stage of evaluation. 


In this final table errors are shown question by question for two groups of 
trainees each numbering 25 again. For simplicity 1 mark has been awarded for 


each correct answer so for 10 questions the group maximum score is 250. 


Group A Group B 


Question 
umber 
: No of Per cent No of Per cent 
errors 
errors 
1 8 32.0 3 12.0 
2 3 12.0 ~ - 
3 * vis = = ‘ 
4 9 36.0 3 12.0 
5 6 24.0 2 8.0 
6 22 88.0 23 92.0 
7 8 32.0 1 4.0 
8 3 12.0 - Pa 
9 19 76-0 2 8.0 
10 2 8.0 1 4.0 
Sa e 
All 
questions 80 32.0 a5 1520 


(group score) 

From this table a number of points emerge. Again we can see that Group B 
achieved a considerably higher success rate indicative of better quality 
training. Question 3 has been correctly answered by all trainees from which it 


may be deduced that it was too elementary. 


Question 6 has been answered equally badly by both groups and the chances are 
either that the question was wrongly worded or that there was a basic fault in 


the training package. 


But perhaps the most interesting feature of the table is that Question 9 has 
been answered wrongly by most of Group A trainees whereas those in the other 
Group had little difficulty. Here it is clear that the fault lies with the 


trainer who obviously gave wrong instructions. 


Assume now that the error is not corrected and that trainees are left with 
entirely the wrong idea. For example, if the question had been - 
Absent household members must be included only at True fete 
their usual place of residence False eS 
(Tick the box which applies) 
- it is clear that an entire group of field staff will fail to enumerate the 
population correctly. The consequence needs little explanation but serves to 


reinforce the need both for training and evaluation. 


It is worth mentioning here that this method of evaluating training can 
sometimes be used in reverse to explain inconsistencies in statistics. 
Reference back to summary assessments could reveal the sort of inaccuracies 
referred to above and help to solve problems and warn the users of census 


Statistics of the dangers of accepting them at face value. 


Part 5 - Conclusion 


This paper has attempted not so much to provide blue print for a training programme 
for Census staff but to stimulate thought and emphasise the need for such a programme, 
Much more could have ben included to explain better the development stages but a good 
deal of this technique is already well known to those experienced in training 


administration. It is not so familiar to Census planners specialising in other 
fields. 


Every census organisation will have a senior planning team comprising experts in 
demography, cartography, communications, data processing as well as administrators 
for financial and organisational needs. Not all, regrettably accept the need for 
co-ordination of all the demands of the various specialists or that they must, 
eventually, be integrated into a cohesive plan for an efficient data collection 
stagee Census taking over the years has become more and more demanding and its 
complexities both technically and administratively are often daunting. To ensure the 
smooth progression from basic concepts to the practicalities of field work there is a 
definite need for a co-ordinator with sufficient authority to play a full part in the 
development of the Census itself. The co-ordinator would be responsible for field 
methodology and the administration of training and head a team of technical authors 
and training specialists. Responsibilities would include both the development and 


field stages of the training programme as well as evaluation. 


The need for such a post cannot be over emphasised. The cost of a Population Census 
is enormous and the cost of failure must also be considered. Poor quality data 
provides inaccurate, incomplete or misleading statistics possibly causing wrong 
policy and planning decisions to be made at the highest level. The need for a 
training programme is indisputable and no effort should be spared to ensure its 


maximum effectiveness. 
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